-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(v2): generate cache key and hash key #5849
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shall we move cache related code to a new cache folder? We will want to reuse them, and launcher already includes too much.
I feel a little reluctant for skipping unit testing. E.g. the unordered item issue is subtle and may be hard to catch.
v2/component/launcher.go
Outdated
} | ||
|
||
func (l *Launcher) generateFingerPrint(cacheKey cacheKey) (string, error) { | ||
b, err := json.Marshal(cacheKey) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can this guarantee unordered items have a stable order? e.g. different input parameter names
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The json package always orders keys when marshalling. Specifically:
- Maps have their keys sorted lexicographically
- Structs keys are marshalled in the order defined in the struct
FYI: https://stackoverflow.com/questions/18668652/how-to-produce-json-with-sorted-keys-in-go and https://golang.org/pkg/encoding/json/#Marshal.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI, you can use this executor input as reference
pipelines/v2/component/runtime_info_test.go
Line 234 in 039e8a6
want: &pipeline_spec.ExecutorInput{ |
The json marshall package handles the unordered item issue. I don't think we need UT for now to verify the functionality of json marshall package. In addition, same logic are used in KFP V1
I'm fine to add UT if we want to verify the logic of generate cacheKey struct based on the runtime info(adding a UT for this method pipelines/v2/component/launcher.go Line 496 in e43b0e8
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Bobgy The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Description of your changes:
Fixed #5816, generate cache key and hash key
Checklist: